Content-based audio playback emphasis

ABSTRACT

Techniques are disclosed for facilitating the process of proofreading draft transcripts of spoken audio streams. In general, proofreading of a draft transcript is facilitated by playing back the corresponding spoken audio stream with an emphasis on those regions in the audio stream that are highly relevant or likely to have been transcribed incorrectly. Regions may be emphasized by, for example, playing them back more slowly than regions that are of low relevance and likely to have been transcribed correctly. Emphasizing those regions of the audio stream that are most important to transcribe correctly and those regions that are most likely to have been transcribed incorrectly increases the likelihood that the proofreader will accurately correct any errors in those regions, thereby improving the overall accuracy of the transcript.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to the following commonly-owned U.S. patentapplications, hereby incorporated by reference:

U.S. patent application Ser. No. 10/923,517, filed on Aug. 20, 2004,entitled, “Automated Extraction of Semantic Content and Generation of aStructured Document from Speech”; and

U.S. patent application Ser. No. 10/922,513, filed on Aug. 20, 2004,entitled, “Document Transcription System Training.”

BACKGROUND

1. Field of the Invention

The present invention relates to audio playback and, more particularly,to audio playback for use in proofreading draft transcripts of speech.

2. Related Art

It is desirable in many contexts to transcribe human speech. In thelegal profession, for example, transcriptionists transcribe testimonygiven in court proceedings and in depositions to produce a writtentranscript of the testimony. Similarly, in the medical profession,transcripts are produced of diagnoses, prognoses, prescriptions, andother information dictated by doctors and other medical professionals.Transcripts in these and other fields typically need to be highlyaccurate (as measured in terms of the degree of correspondence betweenthe semantic content (meaning) of the original speech and the semanticcontent of the resulting transcript) because of the reliance placed onthe resulting transcripts and the harm that could result from aninaccuracy (such as providing an incorrect prescription drug to apatient). It may be difficult to produce an initial transcript that ishighly accurate for a variety of reasons, such as variations in: (1)features of the speakers whose speech is transcribed (e.g., accent,volume, dialect, speed); (2) external conditions (e.g., backgroundnoise); (3) the transcriptionist or transcription system (e.g.,imperfect hearing or audio capture capabilities, imperfect understandingof language); or (4) the recording/transmission medium (e.g., paper,analog audio tape, analog telephone network, compression algorithmsapplied in digital telephone networks, and noises/artifacts due to cellphone channels).

The first draft of a transcript, whether produced by a humantranscriptionist or an automated speech recognition system, maytherefore include a variety of errors. Typically it is necessary toproofread and edit such draft documents to correct the errors containedtherein. Transcription errors that need correction may include, forexample, any of the following: missing words or word sequences;excessive wording; mis-spelled, -typed, or -recognized words; missing orexcessive punctuation; misinterpretation of semantic concepts (e.g.,mistakenly interpreting an allergy to a particular medication as themedication itself); and incorrect document structure (such as incorrect,missing, or redundant sections, enumerations, paragraphs, or lists).

Although it might be possible for the speaker whose speech istranscribed to proofread the draft transcript merely by reading thetranscript (because the content of the speech may be fresh in thespeaker's mind), any other proofreader must typically listen to arecording of the speech while reading the draft transcript in order toproofread it. Proofreading performed in this way can be tedious,time-consuming, costly, and itself error-prone. What is needed,therefore, are improved techniques for correcting errors in drafttranscripts.

SUMMARY

Techniques are disclosed for facilitating the process of proofreadingdraft transcripts of spoken audio streams. In general, proofreading of adraft transcript is facilitated by playing back the corresponding spokenaudio stream with an emphasis on those regions in the audio stream thatare highly relevant or likely to have been transcribed incorrectly.Regions may be emphasized by, for example, playing them back more slowlythan regions that are of low relevance and likely to have beentranscribed correctly. Emphasizing those regions of the audio streamthat are most important to transcribe correctly and those regions thatare most likely to have been transcribed incorrectly increases thelikelihood that the proofreader will accurately correct any errors inthose regions, thereby improving the overall accuracy of the transcript.

Other features and advantages of various aspects and embodiments of thepresent invention will become apparent from the following descriptionand from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1B are dataflow diagrams of systems for facilitating thecorrection of errors in a draft transcript of a spoken audio streamaccording to embodiments of the present invention;

FIG. 2 is a flowchart of a method that is performed by the playbackemphasis system of FIG. 1 in one embodiment of the present invention toemphasize regions of the audio stream during playback;

FIG. 3 is a flowchart of a method for playing back an audio region inaccordance with a specified emphasis factor according to one embodimentof the present invention;

FIG. 4 is a flowchart of a method for identifying a correctness scorefor a region of an audio stream according to one embodiment of thepresent invention;

FIG. 5 is a flowchart of a method for identifying a relevance score fora region of an audio stream according to one embodiment of the presentinvention; and

FIG. 6 is a flowchart of a method for identifying an emphasis factor toapply to a region of an audio stream based on a correctness score and arelevance score of the region.

DETAILED DESCRIPTION

Referring to FIGS. 1A-1B, dataflow diagrams are shown of systems 100 a-bfor facilitating the correction of errors in a draft transcript 124 of aspoken audio stream 102. In general, each of the systems 100 a-b playsback a modified version 122 of the audio stream 102 to a humanproofreader 126 who also has access to the draft transcript 124. Regionsin the audio stream 102 that are highly relevant (important) or that arelikely to have been transcribed incorrectly in the draft transcript 124are emphasized in the modified version 122 of the audio stream 102 thatis played back to the human proofreader 126. Regions may be emphasized,for example, by playing them back more slowly than regions that are oflow relevance and likely to have been transcribed correctly. Suchemphasis may be achieved, for example, by speeding up the playback ofthe remaining regions (having low relevance and high likelihood ofcorrectness) in the audio stream 102 in comparison to a default playbackrate. As a result, the proofreader's 126 attention is focused on thoseregions of the audio stream 102 that are most important to betranscribed correctly and those regions that are most likely to havebeen transcribed incorrectly, thereby increasing the likelihood that theproofreader 126 will correct any errors in those regions. Furthermore,if emphasis is achieved by speeding up the playback of regions that areirrelevant and likely to have been transcribed correctly, proofreadingmay be performed more quickly than with conventional playback methods,but without sacrificing accuracy.

The two systems 100 a-b differ in that the system 100 a shown in FIG. 1Auses an automatic transcription system 128 a to produce the drafttranscript 124, timing information 130, and alternative hypotheses 134,while the system 100 b shown in FIG. 1B uses a human transcriptionist128 b to produce the draft transcript 124 and an automatic speechrecognizer 132 to produce the timing information 130 and alternativehypotheses 134. Because the operation of the two systems 100 a and 100 bis otherwise similar, the two systems may be referred to hereincollectively as system 100. Similarly, the automatic transcriptionsystem 128 a and human transcriptionist 128 b may be referred to hereincollectively as transcription system 128. Differences between the twosystems will be described when they are relevant.

The audio stream 102 may be any kind of spoken audio stream. The spokenaudio stream 102 may, for example, be dictation by a doctor describing apatient visit. The spoken audio stream 102 may take any form. Forexample, it may be a live audio stream received directly or indirectly(such as over a telephone or IP connection), or an audio stream recordedon any medium and in any format.

The draft transcript 124 may be any document that represents some or allof the content in the spoken audio stream 102. The draft transcript 124may, for example, have been generated by transcription system 128including a human transcriptionist, an automated speech recognizer, orany combination thereof. The draft transcript 124 may have beengenerated using any of the techniques disclosed in the above-referencedpatent application entitled “Automated Extraction of Semantic Contentand Generation of a Structured Document from Speech.” As describedtherein, the draft transcript 124 may be either a literal (verbatim)transcript or a non-literal transcript of the spoken audio stream 102.As further described therein, although the draft transcript 124 may be aplain text document, the draft transcript 106 may also, for example, bea structured document, such as an XML document which delineates documentsections and other kinds of document structure.

The draft transcript 124 may be a structured document containing notonly plain text but also document structures representing semantic andsyntactic concepts, as those terms are defined in the above-referencedpatent application entitled “Automated Extraction of Semantic Contentand Generation of a Structured Document from Speech.” As described inmore detail therein, the term “concept” includes, for example, dates,times, numbers, codes, medications, medical history, diagnoses,prescriptions, phrases, enumerations and section cues. The term“content” is used herein to refer generally to any subset of a document,which may therefore include not only plain text but also representationsof one or more concepts.

The techniques disclosed in the above-referenced patent applicationentitled “Automated Extraction of Semantic Content and Generation of aStructured Document from Speech,” and automatic transcription systemsmore generally, produce timing information 130 that correlates contentwith corresponding regions of the audio stream 102. Such timinginformation 130 may, for example, map each word in the draft transcript124 to a corresponding region in the audio stream 102 representing thatword. The following discussion assumes that such timing information 130is available to the playback emphasis system 100. In the system 100 a ofFIG. 1A, the timing information 130 is produced by the automatictranscription system 128 a during production of the draft transcript124. In the system 100 b of FIG. 1B, the timing information 130 isproduced by the automatic speech recognizer 132 based on the audiostream 102 and the draft transcript 124 produced by the humantranscriptionist 128 b.

Referring to FIG. 2, a flowchart is shown of a method 200 that isperformed by the playback emphasis system 100 in one embodiment of thepresent invention to emphasize regions of the audio stream 102 duringplayback. An audio stream iterator 104 enters a loop over each audioregion A 106 in the audio stream (step 202).

A correctness identifier 108 identifies an estimate C 110 of alikelihood that the audio region A was recognized (transcribed)correctly in the draft transcript 124 (step 204). This estimate isreferred to herein as a “correctness score.” Examples of techniques thatmay be used to generate the correctness score C 110 will be describedbelow with respect to FIG. 4.

A relevance identifier 112 identifies a measure R 114 of potentialrelevance (i.e., importance) of the region A (step 206). This measure isreferred to herein as a “relevance score.” Examples of techniques thatmay be used to generate the relevance score R 114 will be describedbelow with respect to FIG. 5.

An emphasis factor identifier 116 identifies an emphasis factor E 118based on the correctness score C 110 and the relevance score R 114 (step208). Examples of techniques that may be used to generate the emphasisfactor E 118 will be described in more detail below with respect to FIG.6.

An audio playback engine 120 plays back the audio region A 106 inaccordance with the emphasis factor E 118 to produce a region of anemphasis-adjusted audio signal 122 that is played to the humanproofreader 126 (step 210). Note that if the emphasis factor E 118indicates a neutral emphasis, the resulting region of theemphasis-adjusted audio stream 122 may be the same as the region A 106of the original audio stream 102. If the audio playback engine 120 is aconventional audio playback engine, a preprocessor (not shown) may applythe emphasis factor E 118 to the audio region A 106 to produce an audiosignal that is suitable for playback by the audio playback engine 120.Furthermore, the emphasis-adjusted audio stream 122 may be furtherprocessed (such as by further speeding it up or slowing it down) inaccordance with user preferences or other requirements. The method 200repeats steps 204-210 for the remaining regions in the audio stream 102(step 212), thereby applying any appropriate emphasis to the regionswhen playing them back to the proofreader 126.

Having described one embodiment of the present invention in general,particular embodiments of the present invention will now be described inmore detail. One way in which regions of the audio stream 102 may beemphasized is by playing them back more slowly than other regions in theaudio stream. The emphasis factor 118 may, therefore, be a timescaleadjustment factor which may be multiplied by a default playback rate toachieve the rate at which to play back the corresponding audio region A106. The audio playback engine 120 may perform this timescale adjustmentin accordance with the emphasis factor 118 when generating theemphasis-adjusted audio signal 122, which is, in this case, atimescale-adjusted version of the audio region A 106.

For example, referring to FIG. 3, a flowchart is shown of a method thatmay be used to implement step 210 of the method 200 shown in FIG. 2 forplaying back the audio region A 106 with any emphasis specified by theemphasis factor E 118. The method identifies a default playback rateP_(D) (step 304). The default playback rate P_(D) may be any playbackrate, such as a real-time playback rate, at which the audio stream 102is to be played back without emphasis. The method identifies anemphasized playback rate P_(E) by dividing the default playback rateP_(D) by the emphasis factor E 118 (step 306). The method plays back theaudio region A at the emphasized playback rate P_(E) (step 308).

Note that since the emphasis factor E may be less than one, equal toone, or greater than one, the “emphasized” playback rate P_(E) may beslower than, faster than, or equal to the default playback rate P_(D).Therefore, although P_(E) is referred to herein as an “emphasized”playback rate, playing back the audio region A 106 at rate P_(E) mayeither emphasize, de-emphasize, or place no emphasis on audio region A106 depending on the value of E (and hence the value of P_(E)). The sameis true generally of techniques other than timescale adjustment that maybe used to modify the audio region A 106 during playback based on theemphasis factor E 118.

Furthermore, an emphasized audio region may be played back at slowerspeeds than other regions in essentially two ways: (1) by decreasing theplayback rate of the emphasized audio region relative to the defaultplayback rate P_(D); and (2) by increasing the playback rate of theregion(s) surrounding the emphasized audio region relative to thedefault playback rate P_(D). Both such techniques are within the scopeof the present invention, and the two may be combined with each other invarious ways. The same is true generally of techniques other thantimescale adjustment that may be used to modify the audio region A 106during playback based on the emphasis factor E 118. One advantage,however, of emphasizing a particular audio region by speeding up theplayback of surrounding audio regions is that doing so decreases thetotal time required to play back the audio stream 102 to the proofreader126, thereby increasing the rate at which proofreading may be performed.

It was stated above that the correctness identifier 108 identifies thecorrectness score 108 for audio region A 106. The correctness identifier108 may identify the correctness score 108 in any of a variety of ways.For example, referring to FIG. 4, a flowchart is shown of a method thatmay be used to implement step 204 of the method 200 shown in FIG. 2 foridentifying the correctness score C 110.

The correctness identifier 108 identifies a prior likelihood ofcorrectness Cp of the region of the draft transcript 124 thatcorresponds to the audio region A 106 (step 402). This region of thedraft transcript 124 may include any kind of “content” as that term isdefined herein. A “prior likelihood of correctness” is any estimate of alikelihood of correctness that is pre-assigned to particular content.For example, human transcriptionists often mistake the words “ascending”and “descending” for each other. Therefore, the words “ascending” and“descending” in the draft transcript 124 are likely to have beenincorrectly transcribed. Such words may be assigned a relatively lowprior likelihood of correctness. Similarly, automatic transcriptionsystems may systematically misrecognize certain words, which may beassigned a relatively low prior likelihood of correctness. Automatictranscription systems often misrecognize different words than humantranscriptionists, and as a result the same word may have a differentprior likelihood of correctness depending on the transcription method inuse.

The correctness identifier 108 identifies a value C_(A) characterizing afeature of the spoken audio stream 102, such as an identity of a speakerof the spoken audio stream 102 or a signal-to-noise ratio of the spokenaudio stream 102 (step 404). If, for example, a particular speaker isknown to be difficult to understand and therefore likely to betranscribed incorrectly, the correctness identifier 108 may assign arelatively low value to C_(A). If, for example, the audio stream 102 hasa relatively high signal-to-noise ratio, then the draft transcript 124was relatively likely to have been transcribed correctly, and thecorrectness identifier 108 may assign a relatively high value to C_(A).

Automatic speech recognizers typically generate a confidence measure foreach word in a document representing a degree of confidence that theword was recognized correctly, i.e., that the word correctly representsthe corresponding speech in the audio stream from which it wasrecognized. If the correctness identifier 108 has access to suchconfidence measures, the correctness identifier 108 may identify a valueC_(M) based on the confidence measure associated with the region of thedraft transcript 124 that corresponds to region A 106 of the audiostream 102 (step 406).

The correctness identifier 108 identifies the overall correctness scoreC 110 based on the individual scores C_(P), C_(A), and C_(M) (step 408).The correctness identifier 108 may, for example, identify the overallcorrectness score C 110 as a weighted sum of C_(P), C_(A), and C_(M).Such a weighting may, for example, favor the emphasis of audio regionshaving a low prior likelihood of correctness, audio streams havingcharacteristics (such as low signal-to-noise ratios) indicating a highlikelihood of error, and regions having low confidence measures.Alternatively, the correctness identifier 108 may identify the overallcorrectness score C 110 as the minimum of C_(P), C_(A), and C_(M). Theseare merely examples; the correctness identifier 108 may identify theoverall correctness score C 110 in any way, such as by using any rule oralgorithm.

Furthermore, the individual scores C_(P), C_(A), and C_(M) are merelyexamples of the factors that the correctness identifier 108 may takeinto account when generating the correctness score 110. The correctnessidentifier 108 may take into account any combination of these or otherfactors, using any weighting or other combination function, whengenerating the correctness score.

It was stated above that the relevance identifier 112 generates therelevance score 114 for audio region A 106. The relevance identifier 112may generate the relevance score 114 in any of a variety of ways. Forexample, referring to FIG. 5, a flowchart is shown of a method that maybe used to implement step 206 of the method 200 shown in FIG. 2 forgenerating the relevance score R 114.

The relevance identifier 112 identifies a prior relevance R_(P) of theregion of the draft transcript 124 corresponding to region A 106 of theaudio stream 102 (step 502). For example, in a medical report, thesection describing allergies of the patient is always highly important(relevant). Therefore, the allergies section may be assigned a highprior relevance. Similarly, certain content, such as the words “no” and“not,” may be assigned a high prior relevance. Furthermore, empty text(which likely represents periods of silence or non-speech events such ascoughs) may be assigned a low prior relevance.

Automatic speech recognizers typically generate a set of alternativehypotheses 134 (i.e., candidate words) for each recognized region in anaudio stream. For example, when the automatic transcription system 128 aattempts to recognize the spoken word “knot,” the system 128 a maygenerate a list of alternative hypotheses 134 consisting of the words“knot,” “not,” “naught,” and “nit,” in that order. The system 128 atypically associates a confidence measure with each hypothesisrepresenting a degree of confidence that the hypothesis accuratelyrepresents the corresponding audio region. The final output of anautomatic speech recognizer, such as the draft transcript 124, typicallyonly includes the best hypothesis (i.e., the hypothesis having thehighest confidence measure) for each corresponding region in the audiostream 102. If, however, the draft transcript 124 includes informationabout competing hypotheses, or if the relevance identifier 112 otherwisehas access to the competing hypotheses 134, the relevance identifier 112may use such competing hypothesis information 134 to generate therelevance score R 114.

For example, the relevance identifier 112 may identify the priorrelevance R_(H) of the competing hypothesis having the highest priorrelevance of all competing hypotheses for the current document region(step 504). In the example above, in which the competing hypotheses are“knot,” “not,” “naught,” and “nit,” the word “not” most likely has thehighest prior relevance. In such a case, the relevance identifier 112may use the prior relevance of the word “not” as the value of R_(H) eventhough the word “not” does not appear in the draft transcript 124.Elevating the relevance of the word “knot” in this way may be usefulbecause it is important to bring the word to the attention of theproofreader 126 in the event that the highly-relevant word “not” wasmisrecognized as “knot.”

The relevance identifier 112 identifies the overall relevance score R114 based on the individual scores R_(P) and R_(H) (step 506). Therelevance identifier 112 may, for example, identify the overallrelevance score R 112 as a weighted sum of R_(P) and R_(H). Such aweighting may, for example, favor the emphasis of audio regions having ahigh prior relevance and having competing hypotheses with a high priorrelevance. This is merely an example; the relevance identifier 112 mayidentify the overall relevance score R 112 in any way. Furthermore, theindividual scores R_(P) and R_(H) are merely examples of the factorsthat the relevance identifier 112 may take into account when generatingthe relevance score 114. Furthermore, the relevance identifier 112 maytake into account any combination of these or other factors, using anyweighting or other combination function, when generating the relevancescore 114. For example, the relevance identifier 112 may identify theoverall relevance score R 114 as the maximum of R_(P) and R_(H).

It was stated above that the emphasis factor identifier 116 generatesthe emphasis factor E 118 based on the correctness score C 110 and therelevance score R 114. The emphasis factor identifier 116 may identifythe emphasis factor E 118 in any of a variety of ways. For example,referring to FIG. 6, a flowchart is shown of a method that may be usedto implement step 208 of the method 200 shown in FIG. 2 for identifyingthe emphasis factor E 118. In the method shown in FIG. 6, the emphasisfactor identifier 116 generates the emphasis factor 118 as a weightedsum of the correctness score 110 and the relevance score 114.

The emphasis factor identifier 116 identifies a weight W_(C) for thecorrectness score C (step 602) and a weight W_(R) for the relevancescore R (step 604). The emphasis factor identifier 116 identifies theemphasis factor E 118 as a weighted sum of C and R, using the weightsW_(C) and W_(R), respectively (step 606). Note that each of the weightsW_(C) and W_(R) may be positive, negative, or equal to zero.

The relevance score R may, for example, be one of the following symbolicvalues: (1) “filler,” corresponding to audio regions (such as silenceand coughs) not having any speech content; (2) “non-transcribed,”corresponding to audio regions containing speech (such as greetings andintermittent conversation with third parties) that is completelyirrelevant and therefore not transcribed; (3) “normal,” corresponding toaudio regions containing normal speech suitable for transcription; and(4) “critical,” corresponding to audio regions containing critical(highly relevant) speech (such as “no” and “not”). Such symbolic valuesmay be ordered, with “filler” representing the lowest relevance and“critical” representing the highest relevance.

One way to use such symbolic values to adjust the playback rate is toassociate a constant playback rate multiplier with each symbolicrelevance value, with lower multipliers being associated with morerelevant content. “Filler” audio regions may be treated as specialcases. Each such region may be played back in a fixed duration (e.g., 1second), or in a duration that is equal to a fixed value (e.g., 1second) plus a fraction (e.g., 1/10) of the original duration of theaudio region. The intent of such a scheme is to play back content at aspeed that is inversely related to its relevance, with the exception of“filler” content, which is played back at a highly accelerated speedthat still allows the user to identify non-filler audio and therebydetermine that the content was incorrectly classified as “filler.”

The correctness score C 110 and the relevance score R 114 may, forexample, be combined to produce the emphasis factor E 118 as follows.The speech recognizer may be assigned a default correctness score C_(R)based on the recognizer's observed average rate of recognition accuracy.Recall that a confidence measure C_(M) is associated with each documentregion. The final correctness score C of a document region may becalculated as C_(M)/C_(R). The final emphasis factor E 118 may beobtained as R/C.

Upper and lower bounds may be imposed on the emphasis factor E. Forexample, if E is a playback rate adjustment factor, it may be limited tothe range [1, 10] to ensure that the audio stream is played back at noless than half the default rate and at no more than twice the defaultrate.

Among the advantages of the invention are one or more of the following.Embodiments of the present invention facilitate the process ofproofreading draft transcripts by playing back the corresponding spokenaudio stream with an emphasis on critical regions in the audio stream.Critical regions are identified based on their content. Morespecifically, a region may be considered critical if it is highlyrelevant or likely to have been transcribed incorrectly. Emphasizingthese regions focuses the attention of the proofreader on them, therebyincreasing the likelihood that the proofreader will correct any errorsin those regions.

As described above, critical regions of the audio stream may beemphasized by playing them back more slowly than non-critical regions.If emphasis is achieved by speeding up the playback of the non-criticalregions relative to a default playback rate, proofreading may beperformed more quickly than with conventional playback methods, butwithout sacrificing accuracy. Furthermore, if emphasis is achieved byslowing down the playback of the critical regions relative to thedefault playback rate, the proofreader may better be able to discern thespeech in those regions and thereby be better able to correct anycorresponding transcription errors. If emphasis is achieved both byspeeding up the playback of the non-critical regions and by slowing downthe playback of the critical regions, it may be possible for the entireaudio stream to be played back in a shorter period of time than if ithad been played back at the default (e.g., real-time) rate, while stillproviding the proofreader with the benefit of slowed-down criticalregions.

Increased speed may be achieved without sacrificing accuracy, and evenwhile improving accuracy, because the de-emphasized non-critical regionsare those which are least likely to correspond to a document regioncontaining an error. Such regions do not require the focused attentionof the proofreader because the corresponding document regions areunlikely to require correction. If de-emphasis is achieved by increasingthe playback rate, such regions can be played back more quickly andthereby decrease the overall time required for proofreading withoutsacrificing accuracy.

Furthermore, embodiments of the present invention do not prevent errorsin non-critical regions from being corrected. Even when a non-criticalregion is de-emphasized, the proofreader may still recognize an error inthe region and correct it. If, for example, a non-critical region isplayed more quickly than usual, the speech in the non-critical regionmay still be audible to the proofreader, who may still recognize anerror in the region and correct it. This feature provides a degree ofprotection against misclassification of regions as non-critical, byeffectively enabling the human proofreader to override such aclassification in the event of an error that is detectable even whenun-emphasized or de-emphasized. This ability to override theclassification of regions as non-critical is absent in prior art systemsthat merely remove or suppress the playback of audio that is classifiedas non-critical.

Although the previous discussion may refer to “critical” and“non-critical” regions, and to “emphasizing” and “de-emphasizing” suchregions, embodiments of the present invention are not limited to suchbinary distinctions between regions and the emphasis placed on them.Rather, any region may be classified as falling along a continuum ofcriticality having a corresponding degree of emphasis falling along itsown continuum. As described above, the correctness score C and relevancescore R, and their respective weights, may have any values, and may becombined in any way to produce the emphasis factor E. The modified audiostream 122 that results from applying the emphasis factor E 118 to theoriginal audio region A 106 may therefore have any degree of emphasis.This flexibility allows the system 100 to emphasize different regions ofthe audio stream 102 to different degrees. When emphasis is achieved byplaying back audio regions more slowly than surrounding regions, theability to provide varying degrees of emphasis in this way produces themost efficient playback speed and the one that is most likely to producethe highest accuracy rate for the amount of time that is required toproofread the draft transcript 124.

Despite this flexibility, embodiments of the present invention may usequantized degrees of emphasis. For example, the system 100 may quantizethe emphasis factor E 118 into three values: emphasized, de-emphasized,and neutral. If emphasis is achieved using timescale adjustment, thesethree values may correspond to a slower-than-realtime playback speed, afaster-than-realtime playback speed, and a realtime playback speed. Thisis merely one example of a way in which emphasis factors may bequantized and does not constitute a limitation of the present invention.

It is to be understood that although the invention has been describedabove in terms of particular embodiments, the foregoing embodiments areprovided as illustrative only, and do not limit or define the scope ofthe invention. Various other embodiments, including but not limited tothe following, are also within the scope of the claims. For example,elements and components described herein may be further divided intoadditional components or joined together to form fewer components forperforming the same functions.

As described above, the playback rate of regions in the audio stream 102may be modified to provide appropriate emphasis. Such playback rateadjustments may be performed with or without additional adjustments,such as pitch adjustment, adjustments to the signal power of thecontained signal, or using perceptually motivated transformations thatshorten vowel playback more than that of consonants.

Placing emphasis on a word may produce a jarring effect to the listener,making the word be difficult to understand and sound unnatural. Such aneffect may be produced, for example, if the playback rate is adjustedsharply to play back an individual word very quickly in comparison tothe words preceding and following it. To address this problem anemphasized word may, for example, be made to sound more natural bygradually increasing the emphasis of speech beginning a few words beforethe emphasized word, and then gradually decreasing the emphasis ofspeech for a few words after the emphasized word. Such smoothing ofemphasis may not only make the emphasized speech sound more natural butalso make it easier to understand, thereby increasing the effectivenessof the emphasized word for correcting transcription errors in the drafttranscript 124.

Similarly, if a word's correctness score is relatively low (and itslikelihood of incorrectness is therefore relatively high), then one ormore subsequent words may be played back slowly to provide the humanproofreader 126 with sufficient time to edit the (likely) incorrectword. Decreasing the playback rate of such subsequent words optimizesthe editing process itself by making it possible to perform editingwithout stopping, rewinding, and then restarting the playback of theaudio stream 102.

Although in particular examples disclosed herein emphasis is placed onregions of the audio stream 102 by playing back such regions more slowlythan surrounding regions, this is not a limitation of the presentinvention. Emphasis may be achieved in other ways. For example, theregion A 106 of the audio stream 102 may be emphasized by increasing thepower of the emphasis-adjusted audio stream 122 corresponding to theaudio region A 106. Furthermore, additional emphasis may be placed on aregion of the audio stream 102 by modifying the way in which thecorresponding content in the draft transcript 124 is rendered. Forexample, additional emphasis may be placed on a region in the audiostream 102 by changing the color, font, or font size of thecorresponding word(s) in the draft transcript 124.

The discussion above refers to correctness score C 110 and relevancescore R 114. Such scores may have values measured on any scale. Forexample, the correctness score 110 may have values in the range [0,1]and the relevance score R 114 may have symbolic values as describedabove. Furthermore, a higher value of the correctness score R 114 mayindicate either a higher likelihood of correctness or a higherlikelihood of error. Therefore, the “correctness” score C 110 may beinterpreted either as a correctness score or as an incorrectness (error)score. Similarly, a higher value of the relevance score R 114 mayindicate either a higher relevance or a lower relevance. Therefore, the“relevance” score R 114 may be interpreted either as a relevance scoreor as an irrelevance score.

Similarly, the emphasis factor E 118 may have a value that is measuredon any scale. Furthermore, a higher value of the emphasis factor E 118may indicate either greater or lesser emphasis. Therefore, the“emphasis” factor E 118 may be interpreted either as an emphasis factoror as a de-emphasis factor.

The discussion above may refer to audio regions that are “highly”relevant and/or “likely” to have been transcribed incorrectly. These andother similar terms are used merely for purposes of illustration and donot impose any limitations on embodiments of the present invention. Forexample, it is not required that an audio region exceed any particularthreshold of relevance or likelihood of error to be emphasized duringplayback. Rather, as the discussion above makes clear, there may be anyrelationship between the correctness score, relevance score, andemphasis factor associated with a particular audio region. In general,the emphasis factor need only be based on the correctness score and/orthe relevance score.

Although in various examples described above the emphasis factoridentifier 116 identifies the emphasis factor E 118 based on acombination of the correctness score C 110 and the relevance score R114, this is not a requirement of the present invention. Rather, theemphasis factor identifier 116 may identify the emphasis factor E 114based solely on the correctness score C 110 or based solely on therelevance score R 114.

Although the modified version of the audio region A 106 may be referredto herein as an “emphasized” or “emphasis-adjusted” audio stream 122,this does not mean that the emphasis-adjusted audio stream 122 mustdiffer from the original audio region A 106. Rather, the“emphasis-adjusted” audio stream 122 may be an emphasized version of theaudio region A 106, a de-emphasized version of the audio region A 106,or the same as the audio region A 106, depending on the value of theemphasis factor E 118.

Furthermore, the term “emphasis” is used herein generally to refer tothe effect of emphasizing the playback of a particular audio region in aparticular context, not specifically to any particular technique forachieving such emphasis. For example, an audio region may be emphasizedby slowing down its playback, by speeding up the playback of surroundingaudio regions, or by a combination of both. It is possible, therefore,to “emphasize” the playback of an audio region by playing back the audioregion itself without modification and by modifying the playback ofsurrounding audio regions. References to “emphasizing” audio regionsherein should be understood to refer to any kind of technique forachieving emphasis.

Although certain embodiments of the present invention disclosed hereindetect and correct errors in documents generated based on speech,techniques disclosed herein may also be applied to detect and correcterrors in documents that were not generated based on speech. Forexample, techniques disclosed herein may be used to identify emphasisfactors for regions in a document and to “play” the regions of thedocument in accordance with the emphasis factors using a text-to-speechengine. A document may be “played” in this way over a telephoneinterface, for example, to minimize the playback time.

Although the preceding description describes factors that may influencethe prior likelihood of correctness and prior relevance of regions inthe draft transcript 124, such factors are merely examples and do notconstitute limitations of the present invention. Examples of otherfactors that may influence the prior likelihood of correctness and/orprior relevance of a region in the draft transcript 124 include theidentity of the speaker, the domain of the audio stream 102 (e.g.,medical or legal), the worktype of the draft transcript 124 (e.g.,letter, discharge summary, progress note, consultation note, dischargesummary, or radiology report in the context of medical reports), and thesection of the draft transcript 124 in which the region occurs. As aresult, the same word may, for example, have a different priorlikelihood of correctness and/or prior relevance depending on thesection of the document in which the word occurs.

The techniques described above may be implemented, for example, inhardware, software, firmware, or any combination thereof. The techniquesdescribed above may be implemented in one or more computer programsexecuting on a programmable computer including a processor, a storagemedium readable by the processor (including, for example, volatile andnon-volatile memory and/or storage elements), at least one input device,and at least one output device. Program code may be applied to inputentered using the input device to perform the functions described and togenerate output. The output may be provided to one or more outputdevices.

Each computer program within the scope of the claims below may beimplemented in any programming language, such as assembly language,machine language, a high-level procedural programming language, or anobject-oriented programming language. The programming language may, forexample, be a compiled or interpreted programming language.

Each such computer program may be implemented in a computer programproduct tangibly embodied in a machine-readable storage device forexecution by a computer processor. Method steps of the invention may beperformed by a computer processor executing a program tangibly embodiedon a computer-readable medium to perform functions of the invention byoperating on input and generating output. Suitable processors include,by way of example, both general and special purpose microprocessors.Generally, the processor receives instructions and data from a read-onlymemory and/or a random access memory. Storage devices suitable fortangibly embodying computer program instructions include, for example,all forms of non-volatile memory, such as semiconductor memory devices,including EPROM, EEPROM, and flash memory devices; magnetic disks suchas internal hard disks and removable disks; magneto-optical disks; andCD-ROMs. Any of the foregoing may be supplemented by, or incorporatedin, specially-designed ASICs (application-specific integrated circuits)or FPGAs (Field-Programmable Gate Arrays). A computer can generally alsoreceive programs and data from a storage medium such as an internal disk(not shown) or a removable disk. These elements will also be found in aconventional desktop or workstation computer as well as other computerssuitable for executing computer programs implementing the methodsdescribed herein, which may be used in conjunction with any digitalprint engine or marking engine, display monitor, or other raster outputdevice capable of producing color or gray scale pixels on paper, film,display screen, or other output medium.

What is claimed is:
 1. A method performed by a computer processorexecuting computer program instructions tangibly stored on a firstcomputer-readable medium to perform a method comprising: (A) deriving,from a region of a document and a corresponding region of a spoken audiostream, a likelihood score representing a likelihood that the region ofthe document correctly represents content in the corresponding region ofthe spoken audio stream, and tangibly storing a representation of thelikelihood score in a second computer-readable medium; (B) selecting arelevance score representing a measure of relevance of the region of thespoken audio stream, the measure of relevance representing a measure ofimportance that the region of the spoken audio stream be brought to theattention of a human proofreader, and tangibly storing a representationof the relevance score in a third computer-readable medium; and (C)deriving, by performing a calculation on the likelihood score and therelevance score, an emphasis factor for modifying emphasis placed on theregion of the spoken audio stream when played back, and storing arepresentation of the emphasis factor in a fourth computer-readablemedium.
 2. The method of claim 1, wherein (C) comprises deriving anemphasis factor for increasing the emphasis placed on the region of thespoken audio stream when played back.
 3. The method of claim 1, wherein(C) comprises deriving an emphasis factor for decreasing the emphasisplaced on the region of the spoken audio stream when played back.
 4. Themethod of claim 1, wherein (C) comprises deriving, from the likelihoodand the measure of relevance, a timescale adjustment factor foradjusting a playback rate of the region of the spoken audio stream. 5.The method of claim 1, wherein (C) comprises deriving, from thelikelihood and the measure of relevance, a signal power adjustmentfactor for adjusting a signal power of the region of the spoken audiostream.
 6. The method of claim 1, further comprising: (D) modifying anemphasis of the region of the spoken audio stream in accordance with theemphasis factor to produce an emphasis-adjusted audio stream.
 7. Themethod of claim 6, further comprising: (E) modifying an emphasis of theregion of the document in accordance with the emphasis factor to producean emphasis-adjusted document region.
 8. The method of claim 6, furthercomprising: (E) playing the emphasis-adjusted audio stream.
 9. Themethod of claim 8, further comprising: (F) correcting errors in thedocument based on the emphasis-adjusted audio stream.
 10. The method ofclaim 6, further comprising: (E) modifying an emphasis of a regionadjacent to the region of the spoken audio stream by an amount less thanthat specified by the emphasis factor.
 11. The method of claim 1,wherein (A) comprises: (A)(1) deriving the likelihood from a priorlikelihood of correctness of the region of the document.
 12. The methodof claim 1, wherein (A) comprises: (A)(1) deriving the likelihood from afeature of the spoken audio stream.
 13. The method of claim 12, whereinthe feature comprises an identity of a speaker of the spoken audiostream.
 14. The method of claim 12, wherein the feature comprises asignal-to-noise ratio of the spoken audio stream.
 15. The method ofclaim 1, wherein (A) comprises: (A)(1) deriving the likelihood from aconfidence measure representing a degree of confidence that the regionof the document correctly represents the content in the correspondingregion of the spoken audio stream, wherein the confidence measure isprovided by an automatic transcription system that produced the regionof the document based on the region of the spoken audio stream.
 16. Themethod of claim 15, wherein (A)(1) comprises deriving the likelihoodfrom the confidence measure, a prior likelihood of correctness of theregion of the document, and a feature of the spoken audio stream. 17.The method of claim 1, wherein (A) comprises: (A)(1) deriving thelikelihood from a prior likelihood of correctness of the region of thedocument.
 18. The method of claim 1, wherein (B) comprises: (B)(1)selecting a prior relevance of the region of the document; and (B)(2)selecting the measure of relevance of the region of the spoken audiostream based on the identified prior relevance of the region of thedocument.
 19. The method of claim 18, wherein (B)(1) comprises selectingthe prior relevance of the region of the document as a relatively highprior relevance if the region of the document contains content in apredetermined set of highly-relevant content.
 20. The method of claim 1,wherein (B) comprises selecting the measure of relevance of the regionof the spoken audio stream as a relatively low relevance if the regionof the spoken audio stream contains no speech.
 21. The method of claim1, wherein the region of the document comprises a hypothesis generatedby an automatic transcription system for the corresponding region of thespoken audio stream, and wherein (B) comprises: (B)(1) identifying acompeting hypothesis generated by the automatic transcription system forthe corresponding region of the spoken audio stream; (B)(2) selecting aprior relevance of the competing hypothesis; and (B)(3) selecting themeasure of relevance based on the prior relevance of the competinghypothesis.
 22. The method of claim 21, wherein (B)(3) comprisesselecting the measure of relevance based on the prior relevance of thecompeting hypothesis and a prior relevance of the region of thedocument.
 23. The method of claim 1, wherein (C) comprises: (C)(1)identifying a rule for identifying the emphasis factor based on theidentified likelihood and the identified measure of relevance; and(C)(2) applying the rule to the identified likelihood and the identifiedmeasure of relevance to derive the emphasis factor.
 24. The method ofclaim 1, further comprising: (A′) prior to (A), generating the documentbased on the spoken audio stream.
 25. The method of claim 24, wherein(A′) comprises using an automated transcription system to generate thedocument based on the spoken audio stream.
 26. The method of 1, furthercomprising: (D) before (C), identifying a default playback rate of thespoken audio stream; and (E) identifying an emphasized playback rate ofthe spoken audio stream as a quotient of the default playback rate andthe emphasis factor.
 27. The method of claim 26, further comprising: (F)playing the spoken audio stream at the emphasized playback rate.
 28. Anapparatus comprising a computer-readable medium tangibly storinginstructions executable by a computer processor to perform a methodcomprising: (A) deriving, from a region of a document and acorresponding region of a spoken audio stream, a likelihood scorerepresenting a likelihood that the region of the document correctlyrepresents content in the corresponding region of the spoken audiostream; (B) selecting a relevance score representing a measure ofrelevance of the region of the spoken audio stream, the measure ofrelevance representing a measure of importance that the region of thespoken audio stream be brought to the attention of a human proofreader;and (C) deriving, by performing a calculation on the likelihood scoreand the relevance score, an emphasis factor for modifying emphasisplaced on the region of the spoken audio stream when played back. 29.The apparatus of claim 28, wherein (C) comprises deriving an emphasisfactor for increasing the emphasis placed on the region of the spokenaudio stream when played back.
 30. The apparatus of claim 28, wherein(C) comprises deriving an emphasis factor for decreasing the emphasisplaced on the region of the spoken audio stream when played back. 31.The apparatus of claim 28, wherein (C) comprises deriving, from thelikelihood and the measure of relevance, a timescale adjustment factorfor adjusting a playback rate of the audio stream.
 32. The apparatus ofclaim 28, wherein (C) comprises deriving, from the likelihood and themeasure of relevance, a signal power adjustment factor for adjusting asignal power of the region of the spoken audio stream.
 33. The apparatusof claim 28, wherein the method further comprises: (D) modifying anemphasis of the region of the spoken audio stream in accordance with theemphasis factor to produce an emphasis-adjusted audio stream.
 34. Theapparatus of claim 33, wherein the method further comprises: (E)modifying an emphasis of the region of the document in accordance withthe emphasis factor to produce an emphasis-adjusted document region. 35.The apparatus of claim 33, further comprising: an audio playback engineplay the emphasis-adjusted audio stream.
 36. The apparatus of claim 35,wherein the method further comprises: (E) correcting errors in thedocument based on the emphasis-adjusted audio stream.
 37. The apparatusof claim 33, wherein the method further comprises: (E) modifying anemphasis of a region adjacent to the region of the spoken audio streamto a lesser extent than specified by the emphasis factor.
 38. Theapparatus of claim 33, wherein (A) comprises: (A)(1) deriving thelikelihood from a prior likelihood of correctness of the region of thedocument.
 39. The apparatus of claim 28, wherein (A) comprises: (A)(1)deriving the likelihood from a feature of the spoken audio stream. 40.The apparatus of claim 28, wherein (B) comprises: (B)(1) identifying aprior relevance of the region of the document; and (B)(2) selecting themeasure of relevance of the region of the spoken audio stream based onthe identified prior relevance of the region of the document.
 41. Theapparatus of claim 28, wherein (C) comprises: (C)(1) identifying a rulefor identifying the emphasis factor based on the identified likelihoodand the identified measure of relevance; and (C)(2) applying the rule tothe identified likelihood and the identified measure of relevance toidentify the emphasis factor.
 42. The apparatus of claim 28, wherein themethod further comprises: (D) generating the document based on thespoken audio stream.
 43. The apparatus of 28, wherein the method furthercomprises: (D) before (C), identifying a default playback rate of thespoken audio stream; and (E) identifying an emphasized playback rate ofthe spoken audio stream as a quotient of the default playback rate andthe emphasis factor.
 44. The apparatus of claim 43, wherein the methodfurther comprises: (E) playing the spoken audio stream at the emphasizedplayback rate.